Systems and methods for configurable demodulation

ABSTRACT

Exemplary embodiments are directed to configurable demodulation of image data produced by an image sensor. In some aspects, a method includes receiving information indicating a configuration of the image sensor. In some aspects, the information may indicate a configuration of sensor elements and/or corresponding color filters for the sensor elements. A modulation function may then be generated based on the information. In some aspects, the method also includes demodulating the image data based on the generated modulation function to determine chrominance and luminance components of the image data, and generating the second image based on the determined chrominance and luminance components.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.14/864,554, filed Sep. 24, 2015, now U.S. Pat. No. ______, which claimspriority under 35 U.S.C. §119(e) to U.S. Provisional Application No.62/207,704, filed Aug. 20, 2015, and entitled “UNIVERSAL DEMOSAIC.” Thedisclosure of each of these applications is considered part of thisapplication, and is hereby incorporated by reference in its entirety.

BACKGROUND OF THE INVENTION

Field of the Invention

The present application relates generally to an apparatus and method foruniversal demosaicing of single-plane, color image data produced by adigital image sensor overlaid with a Color Filter Array (CFA) forproducing a color image.

Description of the Related Art

A demosaicing (also de-mosaicing, demosaicking, or debayering) algorithmis a digital image process used to reconstruct a color image from outputfrom an image sensor overlaid with a CFA. The demosaic process may alsobe known as CFA interpolation or color reconstruction. Most moderndigital cameras acquire images using a single image sensor overlaid witha CFA, so demosaicing may be part of the processing pipeline required torender these images into a viewable format. To capture color images,photo sensitive elements (or sensor elements) of the image sensor may bearranged in an array and detect wavelengths of light associated withdifferent colors. For example, a sensor element may be configured todetect a first, a second, and a third color (e.g., red, green and blueranges of wavelengths). To accomplish this, each sensor element may becovered with a single color filter (e.g., a red, green or blue filter).Individual color filters may be arranged into a pattern to form a CFAover an array of sensor elements such that each individual filter in theCFA is aligned with one individual sensor element in the array.Accordingly, each sensor element in the array may detect the singlecolor of light corresponding to the filter aligned with it.

The Bayer pattern has typically been viewed as the industry standard,where the array portion consists of rows of alternating red and greencolor filters and alternating blue and green color filters. Usually,each color filter corresponds to one sensor element in an underlyingsensor element array.

SUMMARY OF THE INVENTION

The systems, methods, and devices of the invention each have severalaspects, no single one of which is solely responsible for its desirableattributes. Without limiting the scope of this invention as expressed bythe claims which follow, some features will now be discussed briefly.After considering this discussion, and particularly after reading thesection entitled “Detailed Description” one will understand how thefeatures of this invention provide advantages.

In one aspect, a demosaicing system for converting image data generatedby an image sensor into an image, is disclosed. The system includes anelectronic hardware processor, configured to receive informationindicating a configuration of sensor elements of the image sensor and aconfiguration of filters for the sensor elements, generate a modulationfunction based on a configuration of sensor elements and theconfiguration of filters, demodulate the image data based on thegenerated modulation function to determine chrominance and luminancecomponents of the image data, and generate the image based on thedetermined chrominance and luminance components.

In some aspects, the electronic hardware processor is further configuredto generate a set of configuration parameters based on the modulationfunction, extract a set of chrominance components from the image datausing the set of configuration parameters, demodulate the chrominancecomponents into a set of baseband chrominance components using the setof configuration parameters, modulate the set of baseband chrominancecomponents to determine a set of carrier frequencies, extract aluminance component from the image data using the set of carrierfrequencies. The image is generated based on the extracted luminancecomponent and the determined set of baseband chrominance components. Theconfiguration of the image sensor may further comprise one or more ofthe following a period of filter elements comprising at least one filterelement, each filter element comprising a spectral range, and the arrayof filter elements comprising a repeating pattern of the period offilter elements, a size of each filter element having a length dimensionand a width dimension that is different than a respective lengthdimension and a respective width dimension of a corresponding sensorelement of the image sensor, and an array of dynamic range sensorelements, each dynamic range sensor element having an integration time,wherein the integration time controls a level of sensitivity of thecorresponding dynamic range sensor element. In some aspects, thedetermination of the modulation function is based on at least one of theperiod of filter elements, the size of each filter element, and thearray of dynamic range sensor elements.

Another aspect disclosed is a method for converting image data generatedby an image sensor into a second image. The method comprises receivinginformation indicating a configuration of sensor elements of the imagesensor and a configuration of filters for the sensor elements,generating a modulation function based on a configuration of sensorelements and the configuration of filters, demodulating the image databased on the generated modulation function to determine chrominance andluminance components of the image data; and generating the second imagebased on the determined chrominance and luminance components. In someaspects, the method also includes generating a set of configurationparameters based on the determined modulation function, extracting a setof chrominance components from the image data using the set ofconfiguration parameters, demodulating the set of chrominance componentsinto a set of baseband chrominance components using the set ofconfiguration parameters, modulating the set of baseband chrominancecomponents to determine a set of carrier frequencies, and extracting aluminance component from the image data using the set of carrierfrequencies, wherein the generation of the second image is based on theextracted luminance component and the set of baseband chrominancecomponents.

In some aspects, the configuration of the image sensor is defined by oneor more of the following: a period of filter elements comprising atleast one filter element, each filter element comprising a spectralrange, and the array of filter elements comprising a repeating patternof the period of filter elements, a size of each filter element having alength dimension and a width dimension that is different than arespective length dimension and a respective width dimension of acorresponding sensor element of the image sensor, and an array ofdynamic range sensor elements, each dynamic range sensor element havingan integration time, wherein the integration time controls a level ofsensitivity of the corresponding dynamic range sensor element. In someaspects, the determination of the modulation function is based on atleast one of the period of filter elements, the size of each filterelement, and the array of dynamic range sensor elements.

Another aspect disclosed is a non-transitory computer-readable mediumcomprising code that, when executed, causes an electronic hardwareprocessor to perform a method of converting image data generated by animage sensor into a second image. The method includes receivinginformation indicating a configuration of sensor elements of the imagesensor and a configuration of filters for the sensor elements,generating a modulation function based on a configuration of sensorelements and the configuration of filters, demodulating the image databased on the generated modulation function to determine chrominance andluminance components of the image data; and generating the second imagebased on the determined chrominance and luminance components. In someaspects, the method further includes generating a set of configurationparameters based on the determined modulation function; extracting a setof chrominance components from the image data using the set ofconfiguration parameters; demodulating the set of chrominance componentsinto a set of baseband chrominance components using the set ofconfiguration parameters; modulating the set of baseband chrominancecomponents to determine a set of carrier frequencies; extracting aluminance component from the image data using the set of carrierfrequencies. The generation of the second image is based on theextracted luminance component and the set of baseband chrominancecomponents.

In some aspects, the configuration of the image sensor is defined by oneor more of the following: a period of filter elements comprising atleast one filter element, each filter element comprising a spectralrange, and the array of filter elements comprising a repeating patternof the period of filter elements, a size of each filter element having alength dimension and a width dimension that is different than arespective length dimension and a respective width dimension of acorresponding sensor element of the image sensor, and an array ofdynamic range sensor elements, each dynamic range sensor element havingan integration time, wherein the integration time controls a level ofsensitivity of the corresponding dynamic range sensor element. In someaspects, the determination of the modulation function is based on atleast one of the period of filter elements, the size of each filterelement, and the array of dynamic range sensor elements.

Another aspect disclosed is a demosaicing apparatus for converting aimage data generated by an image sensor into a second image. Theapparatus includes means for receiving information indicating aconfiguration of sensor elements of the image sensor and a configurationof filters for the sensor elements, means for generating a modulationfunction based on a configuration of sensor elements and theconfiguration of filters, means for demodulating the image data based onthe generated modulation function to determine chrominance and luminancecomponents of the image data; and means for generating an image based onthe determined chrominance and luminance components.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a simplified example of a 2×2 Bayer CFA pattern withRGB spectral components having a 1:1 ratio to the image sensorcomponents.

FIG. 2 illustrates a simplified example of a 3×3 Bayer CFA pattern withRGB spectral components having a 1.5:1 ratio to the image sensorcomponents.

FIG. 3 illustrates a simplified example of a 4×4 Lukac CFA pattern withRGB spectral components having a 1:1 ration to the image sensorcomponents.

FIG. 4 illustrates an example of a Fourier spectrum representation ofFIG. 1.

FIG. 5 illustrates an example of a Fourier spectrum representation ofFIG. 2.

FIG. 6 illustrates an example of a Fourier spectrum representation ofFIG. 3.

FIG. 7 illustrates an example of a Fourier spectrum representation ofFIG. 2 and an example resulting product of a demosaicing process.

FIG. 8 illustrates a simplified example of a process for extractingchrominance components from a Fourier spectrum representation of FIG. 2.

FIG. 9 illustrates a simplified example of a process for demodulating aset of chrominance components to the baseband of the Fourier spectrum.

FIG. 10 illustrates a simplified example of a first step for modulatinga set of baseband chrominance components to acquire a set of associatedcarrier frequencies.

FIG. 11 illustrates a simplified example of a second step for modulatinga set of baseband chrominance components to acquire a set of associatedcarrier frequencies.

FIG. 12 illustrates a simplified process of estimating the luminancechannel in the Fourier spectrum.

FIG. 13A is a flowchart of a method for converting a image datagenerated by an image sensor into a second image.

FIG. 13B is a flowchart of a method for demodulating an image.

FIG. 14A illustrates an embodiment of a wireless device of one or moreof the mobile devices of FIG. 1.

FIG. 14B illustrates an embodiment of a wireless device of one or moreof the mobile devices of FIG. 1.

FIG. 15 is a functional block diagram of an exemplary device that mayimplement one or more of the embodiments disclosed above.

DETAILED DESCRIPTION

The following detailed description is directed to certain specificembodiments of the invention. However, the invention can be embodied ina multitude of different ways. It should be apparent that the aspectsherein may be embodied in a wide variety of forms and that any specificstructure, function, or both being disclosed herein is merelyrepresentative. Based on the teachings herein one skilled in the artshould appreciate that an aspect disclosed herein may be implementedindependently of any other aspects and that two or more of these aspectsmay be combined in various ways. For example, an apparatus may beimplemented or a method may be practiced using any number of the aspectsset forth herein. In addition, such an apparatus may be implemented orsuch a method may be practiced using other structure, functionality, orstructure and functionality in addition to, or other than one or more ofthe aspects set forth herein.

Although the examples, systems, and methods described herein aredescribed with respect to digital camera technologies, they may beimplemented in other imaging technology as well. The systems and methodsdescribed herein may be implemented on a variety of differentphotosensitive devices, or image sensors. These include general purposeor special purpose image sensors, environments, or configurations.Examples of photosensitive devices, environments, and configurationsthat may be suitable for use with the invention include, but are notlimited to, semiconductor charge-coupled devices (CCD) or active sensorelements in CMOS or N-Type metal-oxide-semiconductor (NMOS)technologies, all of which can be germane in a variety of applicationsincluding, but not limited to digital cameras, hand-held or laptopdevices, and mobile devices (e.g., phones, smart phones, Personal DataAssistants (PDAs), Ultra Mobile Personal Computers (UMPCs), and MobileInternet Devices (MIDs)).

The Bayer pattern is no longer the only pattern being used in theimaging sensor industry. Multiple CFA patterns have recently gainedpopularity because of their superior spectral-compression performance,improved signal-to-noise ratio, or ability to provide HDR imaging.

Alternative CFA designs that require modified demosaicing algorithms arebecoming more ubiquitous. New CFA configurations have gained popularitydue to (1) consumer demand for smaller sensor elements, and (2) advancedimage sensor configurations. The new CFA configurations include colorfilter arrangements that break from the standard Bayer configuration anduse colors of a spectrum beyond the traditional Bayer RGB spectrum,white sensor elements, or new color filter sizes. For instance, newcolor filter arrangements may expose sensor elements to a greater rangeof light wavelengths than the typical Bayer RGB configuration, and mayinclude RGB as well as cyan, yellow, and white wavelengths (RGBCYW).Such arrangements may be included in image sensors with sensor elementsof a uniform size. Other arrangements may include a pattern of differentsized sensor elements, and thus, different sized color filters.Furthermore, industry demand for smaller sensor elements is creating anincentive to vary the standard 1:1 color filter to sensor element ratio,resulting in color filters that may overlap a plurality of sensorelements.

Non-Bayer CFA sensors may have superior compression of spectral energy,ability to deliver improved signal-to-noise ratio for low-light imaging,or ability to provide high dynamic range (HDR) imaging. A bottleneck tothe adaption of emerging non-Bayer CFA sensors is the unavailability ofefficient and high-quality color-interpolation algorithms that candemosaic the new patterns. Designing a new demosaic algorithm for everyproposed CFA pattern is a challenge.

Modern image sensors may also produce raw images that cannot bedemosaiced by conventional means. For instance, High Dynamic Range (HDR)image sensors create a greater dynamic range of luminosity than ispossible with standard digital imaging or photographic techniques. Theseimage sensors have a greater dynamic range capability within the sensorelements themselves. Such sensor elements are intrinsically non-linearsuch that the sensor element represents a wide dynamic range of a scenevia non-linear compression of the scene into a smaller dynamic range.

Disclosed herein are methods and systems that provide interpolation andclassification filters that can be dynamically configured to demosaicraw data acquired from a variety of color filter array sensors. The setof interpolation and classification filters are tailored to one or moregiven color filter arrays. In some implementations, the color filterscan be pure RGB or include linear combinations of the R, G, and Bfilters.

The word “exemplary” is used herein to mean “serving as an example,instance, or illustration.” Any embodiment described herein as“exemplary” is not necessarily to be construed as preferred oradvantageous over other embodiments.

The term “direct integration” may include a power or data connectionbetween two or more components (e.g., a processor and an image sensor)over a wired or wireless connection where the two components transferand/or receive data in a direct link.

The term “indirect connection” may include a power or data connectionover an intermediary device or devices between two or more components(e.g., a processor and an image sensor), or a device that may configurethe components, the components having no direct connection to eachother.

The term “substantially” is used herein to indicate within 10% of themeasurement expressed, unless otherwise stated.

The words “color filter array,” “filter array,” and “filter element” arebroad terms and are used herein to mean any form of filtering technologyassociated with filtering spectrums of electromagnetic radiation,including visible and non-visible wavelengths of light.

The term “color filter array” or CFA may be referred to as a “filterarray,” “color filters,” “RGB filters,” or “electromagnetic radiationfilter array.” When a filter is referred to as a red filter, a bluefilter, or a green filter, such filters are configured to allow light topass through that has one or more wavelengths associated with the colorred, blue, or green, respectively.

The term “respective” is used herein to mean the corresponding apparatusassociated with the subject. When a filter is referenced to a certaincolor (e.g., a red filter, a blue filter, a green filter) suchterminology refers to a filter configured to allow the spectrum of thatcolor of light to pass through (e.g., wavelengths of light that aregenerally associated with that color).

FIG. 1 illustrates a first example configuration of a traditional 2×2Bayer CFA pattern 100 using a standard 1:1 size ratio of RGB colorfilter to sensor element. The CFA pattern 100 is a square made up offour smaller squares 101-104, wherein each of the four smaller squares101-104 is representative of both an individual sensor element and anindividual color filter. A first sensor element 101 is labeled with theletter “G” signifying a green color filter overlaying the first sensorelement 101. A second sensor element 102 is labeled with an “R”signifying a red color filter overlaying the second sensor element 102.A third sensor element 103 labeled with the letter “B” signifying a bluecolor filter overlaying the third sensor element 103. A fourth sensorelement 104 labeled again with the letter “G” signifying the green coloroverlaying the fourth sensor element 104.

Image sensor configuration 100 includes color filter elements that havelength and width dimensions that are substantially equal to the lengthand width dimensions of the sensor elements (101, 102, 103, 104).

FIG. 2 illustrates a second example configuration 200 of a 3×3 sensorelement array 205 with a Bayer color filter configuration. The Bayercolor filter configuration 200 includes Bayer color filter elements thatare 1.5 times the sensor element size. The configuration 200 is composedof nine smaller squares outlined with dashed lines, the smaller squaresrepresenting sensor elements in a 3×3 configuration. Overlaying the 3×3sensor element array 205 is a 2×2 pattern of larger squares made up ofsolid lines, each larger square representing a color filter element andlabeled with an alphabetical letter. The first filter element 201labeled “G” allows a spectrum of green light to pass. The second filterelement 202 labeled “R” allows a spectrum of red light to pass. A thirdfilter element 203 labeled “B” allows a spectrum of blue light to pass.A fourth filter element 204 labeled “G” allows a spectrum of green lightto pass.

The filter elements in configuration 200 may have a length and widthdimension that is 1.5× greater than the corresponding length and widthdimension of the sensor element, thus providing a broader spectral rangethan the 2×2 Bayer CFA pattern 100.

FIG. 3 illustrates a third example configuration 300 of a 4×4 sensorelement array with a Lukac pattern using the standard 1:1 size ratio ofRGB color filter to sensor element. The configuration 300 includes up ofsixteen sensor elements 301-316, organized in a 4×4 configuration.Elements 301-316 are labeled with “G”, “R”, or “B”, indicating they areoverlaid with green, red, or blue color filters respectively.

The example configurations of FIGS. 1, 2, and 3 may each be described asa period of filter elements. The periodic arrangement of filter elementsrepresents an irreducible minimum pattern that may be duplicated anumber of times and overlaid upon an image sensor array to create a CFAfor use with (and/or incorporated with) an image sensor. The periodicarrangement of filter elements may comprise one or more filter elements,each filter element having configured to allow a wavelength, or a rangeof wavelengths, of light pass through the filter element.

Information of an image sensor configuration may include a size of eachfilter element in the CFA, periodicity of filter elements, the size ofeach filter element, and/or the size of each sensor element. The eachfilter element can be defined as having a length dimension and a widthdimension. A corresponding sensor element or sensor elements) may have asubstantially identical width and length dimension, or differentdimensions. Additionally, an image sensor may be configured to includean array of dynamic range sensor elements, each dynamic range sensorelement having an integration time where the integration time controlsthe effective sensitivity of the sensor elements to exposed radiation.

FIG. 4 illustrates a single plane spectral image 400 for the firstexample configuration of the traditional 2×2 Bayer CFA pattern 100 usingthe standard 1:1 size ratio of RGB color filter to sensor element,described above. The single pane spectral image 400 may also be referredto in mathematical terms as y[n] throughout this disclosure. The singleplane spectral image 400 is represented by a square 406 of equal lengthand width. The square 406 may represent a frequency plane on a Fourierdomain where the edges of the square 406 are representative of thelimitations of the frequency range for the example 2×2 Bayer CFA pattern100. The frequency range of the square has an x-axis and a y-axisproperty shown by the f_(x) 404 and f_(y) 405 arrows, respectively.

Along the four perimeter edges of the square 406 are example first andsecond chrominance components 401 and 402 of the single plane spectralimage 400. Chrominance components 401 and 402 indicate example areaswhere the chrominance channels exist in the Fourier domain. A luminancecomponent 403 indicates an example area of luminance magnitude in theFourier domain. In this example, the chrominance components 401 402 andluminance components 403 are presented to make identification of thespectral frequency corresponding to the luminance component 403 andchrominance components (401, 402) easily visible. The single planespectral image 400 illustrated may also be referred to as the LC₁C₂domain.

The single plane spectral image 400 of FIG. 4 illustrates an exampleBayer CFA spectrum produced by the 2×2 Bayer CFA pattern 100 discussedabove. FIG. 4 exemplifies how the location and size of the period ofcolor filters relative to the sensor elements that define thisparticular image sensor configuration 100 affect the frequency domainrepresentation of the CFA signal of the output image 400. In this case,the frequency domain representation of the example Bayer CFA spectrum400 comprises a luminance component 403 at the baseband frequency (e.g.,(0, 0)), and a set of first chrominance components 401 and second setchrominance components 402. Here, the luminance component 403 resides inthe baseband of the spatial domain at the spatial frequency (0, 0),while the C1 401 components may reside at the (0, 0.5), (0.5, 0), (0,−0.5), and (−0.5, 0) frequencies and the C2 602 components may reside atthe (−0.5, 0.5), (0.5, 0.5), (0.5, −0.5), and (−0.5, −0.5) frequencies.However, FIG. 4 is just one example, and a variety of image sensorconfigurations may result in raw images with a variety of single planespectral images with a variety of CFA spectrums.

FIG. 5 illustrates an example single plane spectral image 500 derivedfrom the second example configuration 200 having the 3×3 sensor elementarray 205 with a Bayer color filter configuration. The single planespectral image 500 includes a large outer square 504 containing asmaller square 505. The frequency range of the square 504 has an x-axisand a y-axis property shown by the f_(x) 405 and f_(y) 404 arrows,respectively. The large outer square 504 may represent a frequency planeon a Fourier domain where the edges of the square 504 are representativeof the limitations of the frequency range for the example 3×3 sensorelement array 205 with a Bayer color filter configuration. The smallersquare 505 represents the spatial frequency range of the single planespectral image 500 that may contain a first chrominance component 501and a second chrominance component 502 of the single plane spectralimage 500. A luminance component 503 indicates an example area ofluminance magnitude in the Fourier domain. The single plane spectralimage 500 illustrated may also be referred to as the LC₁C₂ domain.

FIG. 5 shows that the luminance component 503 occupies the basebandfrequency range while the first chrominance components 501 and secondchrominance components 502 are modulated at the frequency limitations ofthe smaller square 505. In the case of the 3×3 sensor element array 205with Bayer configured color filters being 1.5 times the size of thesensor elements 200, the chrominance components may be located in thefrequency plane at a spatial frequency range of −0.33 to 0.33. Forexample, the first channel chrominance components 501 may reside at (0,0.33), (0.33, 0), (0, −0.33), and (−0.33, 0) frequencies and the secondchannel chrominance components 502 may reside at the (−0.33, 0.33),(0.33, 0.33), (0.33, −0.33), and (−0.33, −0.33) frequencies. It is notedthat in this single plane spectral image 500 there may existinterference or crosstalk between the luminance component 503 and thechrominance components 501, 502. The crosstalk can be strongest betweenthe luminance component 503 and the first chrominance components 501.

FIG. 6 illustrates an example of a single plane spectral image 600 forthe 4×4 sensor element array 300 with a Lukac pattern using the standard1:1 size ratio of RGB color filter to sensor element, described above.The single plane spectral image 600 is represented with a large outersquare 504 containing a smaller square 505. The smaller square 505represents a spatial frequency range of the single plane spectral image600. Along the four perimeter edges of the square 505 are chrominancecomponents 501 502 of the single plane spectral image 600. Thechrominance components are organized in a hexagonal formation, andrepresented as two color-difference components labeled as C1 601 and C2602. Both horizontally oriented sides, or segments of the smaller squarecontain two chrominance components, both labeled C2 602, with eachcomponent situated toward the ends of the segments. Both verticallyoriented sides, or segments of the smaller square contain onechrominance component, each labeled C1 601, with each component situatedin the middle of the segment. Situated in the middle of the smallersquare is a single circle representing the luminance component 603indicating an example area of magnitude where the luminance component603 exists in the Fourier domain. The single plane spectral image 400illustrated may also be referred to as the LC₁C₂ domain. This circle islabeled with an L. The frequency range of the square has an x-axis and ay-axis property shown by the f_(x) 604 and f_(y) 605 arrows,respectively.

FIG. 6 illustrates that the luminance occupies the baseband while thechrominance is modulated at the frequency limitations of the spatialfrequency range of the single plane spectral image 600, represented bythe smaller square. In some aspects of the configuration of the 4×4sensor element array 300, the chrominance may be located in thefrequency plane at a spatial frequency range of −0.25 to 0.25. Forexample, in some aspects, chrominance component C1 may be modulated atspatial frequencies (0, 0), and the second chrominance component C2 maybe modulated at spatial frequencies (−0.25, 0.25). The single planespectral image 600 includes interference or crosstalk between componentsand the crosstalk may be strongest between the luminance 603 and themodulated chrominance components C1 and C2.

FIG. 7 illustrates demosaicing of the single plane spectral image 500.In this example, the single plane spectral image 500 is processed by amethod 1300, discussed with reference to FIG. 13A below, to produce atriple plane RGB image 700. The demosaiced image 700 that results fromdemosaic method 1300 may include a triple plane RGB image 700, but thisexample should not be seen as limiting. The resulting demosaiced imagemay be any color model (e.g., CMYK) and may exist in a plurality ofspectral planes, or a single plane.

Further to the example in FIG. 7, the demosaic method 1300 generallyuses an image sensor configuration defined by a period of a CFA patternto convert the data points corresponding to the chrominance components401, 402 and luminance component 403 of the single plane spectral image400 produced by the image sensor using that particular CFA pattern.Equation 1 below enables expression of the CFA pattern y[n] in terms ofthe luminance component 403 and chrominance components 401, 402 n=[n₁,n₂] where n represents an address to a spatial coordinate on antwo-dimensional square lattice 404 having a horizontal position (n₁) anda vertical position (n₂). Using the Bayer CFA pattern 100 as an example,a data value at point n can be represented by the following equation:

y[n]=l[n ]+((−1)^(n) ¹ −(−1)^(n) ² )c ₁ [n ]+(−1)^(n) ¹ ^(+n) ² c ₂[n]   (1)

Where:

y[n]: CFA data value at point n=[n₁, n₂],

-   -   l[n]: luminance value at point n=[n₁, n₂],    -   c₁[n]: chrominance value at point n=[n₁, n₂],    -   c₂[n]: chrominance value at point n=[n₁, n₂].        As an example, an LC₁C₂ to RGB transformation of the Bayer CFA        pattern 100 can be given by:

$\begin{matrix}{\begin{bmatrix}L \\C_{1} \\C_{2}\end{bmatrix} = {{{\frac{1}{4}\begin{bmatrix}1 & 2 & 1 \\{- 1} & 0 & 1 \\{- 1} & 2 & {- 1}\end{bmatrix}}\begin{bmatrix}R \\G \\B\end{bmatrix}}.}} & (2)\end{matrix}$

Where:

-   -   L: Luminance component of a single plane spectral image,    -   C₁: First color channel chrominance component of a single plane        spectral image,    -   C₂: Second color channel chrominance component of a single plane        spectral image, and    -   R, G, B: Red, Green, Blue.

Taking the Fourier transform of equation 1, the Bayer CFA pattern 100can be represented in the spectral domain as:

$\begin{matrix}{{Y\left( \underset{\_}{f} \right)} = {{L\left( \underset{\_}{f} \right)} + {C_{1}\left( {\underset{\_}{f} - \begin{bmatrix}\frac{1}{2} \\0\end{bmatrix}} \right)} - {C_{1}\left( {\underset{\_}{f} - \begin{bmatrix}0 \\\frac{1}{2}\end{bmatrix}} \right)} + {{C_{2}\left( {\underset{\_}{f} + \begin{bmatrix}\frac{1}{2} \\\frac{1}{2}\end{bmatrix}} \right)}.}}} & (3)\end{matrix}$

Thus, a spatial domain modulation function ((−1)^(n) ¹ −(−1)^(n) ² )encodes the first channel chrominance component 401, C1 in atwo-dimensional carrier wave with normalized frequencies (½, 0) and (0,½) and another spatial-domain modulation function (−1)^(n) ¹ ^(+n) ²encodes the second channel chrominance component 402, C2 in atwo-dimensional carrier wave with the normalized frequency (½, ½).Deriving the equivalent equations (1) to (3) above for an arbitrary CFApattern is discussed below.

FIG. 8 illustrates an example method for filtering a single planespectral image 500 to extract the chrominance components 501, 502 usinga filter set 800, 801, 802, 803. In this example embodiment, the filterset 800, 801, 802, 803 may be a pair of high-pass filters

$h{\frac{\lambda}{1}\left\lbrack \underset{\_}{n} \right\rbrack}\mspace{14mu} {and}\mspace{14mu} h{\frac{\lambda}{2}\left\lbrack \underset{\_}{n} \right\rbrack}$

adapted to a specific CFA pattern. To extract modulated chrominancecomponents,

$\begin{matrix}{{c{\frac{\lambda}{1m}\left\lbrack \underset{\_}{n} \right\rbrack}}\overset{\Delta}{=}{{m{\frac{\lambda}{c_{1}}\left\lbrack \underset{\_}{n} \right\rbrack}{c_{1}\left\lbrack \underset{\_}{n} \right\rbrack}\mspace{14mu} {and}\mspace{14mu} c{\frac{\lambda}{2m}\left\lbrack \underset{\_}{n} \right\rbrack}}\overset{\Delta}{=}{m{\frac{\lambda}{c_{2}}\left\lbrack \underset{\_}{n} \right\rbrack}{c_{2}\left\lbrack \underset{\_}{n} \right\rbrack}}}} & (4)\end{matrix}$

for each λε{circumflex over (Λ)}_(M)*\(0,0) from a given CFA patterny[n]. In some aspects, the filtering equations may be

$\begin{matrix}{{{\left\lbrack \underset{\_}{n} \right\rbrack} = {\sum\limits_{\underset{\_}{m}}{{y\left\lbrack \underset{\_}{n} \right\rbrack}h{\frac{\lambda}{i}\left\lbrack {\underset{\_}{n} - \underset{\_}{m}} \right\rbrack}}}},{i = 1},2.} & (5)\end{matrix}$

Where:

[n] extracted chrominance component at color channel i of point n,

-   -   y[n] CFA data value at point n=[n₁, n₂],    -   h_(i) ^(λ) [n−m]: high pass filter for point n−m, an address of        a point in the Fourier domain described as a difference used to        index the filter coefficient, indicative of a spatially        invariant filter (i.e., a pattern consistent throughout the        sensor),    -   n: a point that neighbors point m in a first image represented        in a Fourier spectrum, and    -   m: a point in the spectral domain, an integer on a 2d grid        (x,y), the 2d grid being the spectral domain of a Fourier        transform.

In this example, the initial high pass filter (h₁ ⁽¹⁾) may filter ahorizontal set of chrominance components while the proceeding filer (h₁^((M))) may filter a vertical set of chrominance components from thefrequency domain.

FIG. 9 illustrates using a demodulation function to modulate theextracted chrominance components 804, 805, 806, 807 from FIG. 8 intobaseband chrominance components 901, 902, 903, 904. This may beaccomplished by using the analytically derived modulation functions

$m{\frac{\lambda}{c_{1}}\left\lbrack \underset{\_}{n} \right\rbrack}\mspace{14mu} {and}\mspace{14mu} m{\frac{\lambda}{c_{2}}\left\lbrack \underset{\_}{n} \right\rbrack}$

The demodulation operation is described by equation 6 as shown below:

$\begin{matrix}{= \left\{ {\begin{matrix}{\frac{\left\lbrack \underset{\_}{n} \right\rbrack}{m{\frac{\lambda}{c_{i}}\left\lbrack \underset{\_}{n} \right\rbrack}},} & {{m{\frac{\lambda}{c_{i}}\left\lbrack \underset{\_}{n} \right\rbrack}} \neq 0} \\0 & {{m{\frac{\lambda}{c_{i}}\left\lbrack \underset{\_}{n} \right\rbrack}} = 0}\end{matrix},{i = 1},2.} \right.} & (6)\end{matrix}$

Where:

-   -   [n]: extracted chrominance component at point n, where the        channel of the component equals i,    -   m_(C) _(i) ^(λ) [n]: modulation function for a chrominance        channel, the channel being equal to i.

FIG. 9 illustrates the demodulation of the chrominance componentsextracted using the high pass filtering derived from the modulationfunction into a set of baseband chrominance components. As describedabove, the extracted chrominance components comprise the vertical andhorizontal aspects of C1, and the diagonal aspects of C2. Similar to theFourier representation 500 in FIG. 5, the extracted chrominancecomponents are illustrated as four squares 804, 805, 806, 807, eachsquare a Fourier representation of an image produced by the Bayer 3×3sensor element array 200 in FIG. 2. The four squares 804, 805, 806, 807each contain a smaller square 812, 813, 814, 815 respectively, where thesmaller square 812, 813, 814, 815 represents the spatial frequency rangeof the single plane spectral image 500 that may contain the chrominancecomponents of the image. Along two of the four perimeter edges of theinterior smaller square 812 of the first square 804 are two circles 808representing the chrominance components of the single plane spectralimage 500. These circles 808 represent the horizontal C1 components, theC1 components on the left and right sides of the smaller square 812.Along two of the four perimeter edges of the interior smaller square 813of the second square 805 are circles 809 representing the chrominancecomponents of the single plane spectral image 500. These circles 809represent the vertical C1 components, the C1 components on the top andbottom sides of the smaller square 813. Situated upon two of the fourcorners of the interior smaller square 814 of the third square 806 arecircles 810 representing the chrominance components of the single planespectral image 500. These circles 810 represent the diagonal C2components, the C2 components occupying the top right corner and thebottom left corner of the smaller square 814. Situated upon two of thefour corners of the interior smaller square 815 of the fourth square 807are circles 811 representing the chrominance components of the singleplane spectral image 500. These circles 811 represent another set ofdiagonal C2 components, the C2 components occupying the top left cornerand the bottom right corner of the smaller square 815.

FIG. 9 further illustrates the set of baseband chrominance components901, 902, 903, 904 following demodulation of the extracted chrominancecomponents 804, 805, 806, 807, respectively. For example, the firstbaseband chrominance component 905 is represented by a large square 901that houses a smaller square 909. Following a demodulation function 913,the set of chrominance components 808 in the first set of extractedchrominance components 804 are merged into a baseband chrominancecomponent 905. However, in contrast to the corresponding extractedchrominance components 808, the baseband chrominance component containsa single chrominance component 905 residing at the baseband frequency,and labeled as C1, referring to a first color channel chrominancecomponent 905. Similarly, a second baseband chrominance component 906 isrepresented by a large square 902 that houses a smaller square 910.Following a demodulation function 914, the set of chrominance components809 in the first set of extracted chrominance components 805 are mergedinto a baseband chrominance component 906. However, in contrast to thecorresponding extracted chrominance components 809, the basebandchrominance component contains a single chrominance component 906residing at the baseband frequency, and labeled as C1, referring to afirst color channel chrominance component 906.

Further to FIG. 9, a third baseband chrominance component 907 isrepresented by a large square 903 that houses a smaller square 911.Following a demodulation function 915, the set of chrominance components810 in the first set of extracted chrominance components 806 are mergedinto a baseband chrominance component 907. However, in contrast to thecorresponding extracted chrominance components 810, the basebandchrominance component contains a single chrominance component 907residing at the baseband frequency, and labeled as C2, referring to asecond color channel chrominance component 907. Similarly, a fourthbaseband chrominance component 908 is represented by a large square 904that houses a smaller square 912. Following a demodulation function 916,the set of chrominance components 811 in the first set of extractedchrominance components 807 are merged into a baseband chrominancecomponent 908. However, in contrast to the corresponding extractedchrominance components 811, the baseband chrominance component containsa single chrominance component 908 residing at the baseband frequency,and labeled as C2, referring to a second color channel chrominancecomponent 908.

FIG. 10 illustrates an example modulation 917, 918 to merge the multiplebaseband chrominance components 905, 906, 907, 908 into a singlebaseband chrominance component 1005, 1006 for each one of two colorchannels. FIG. 10 includes the set of a first baseband chrominancecomponent 905, a second baseband chrominance component 906, a thirdbaseband chrominance component 907, and a fourth baseband chrominancecomponent 908 described above with respect to FIG. 9, and also includesa set of modulation functions. The baseband chrominance components ofthe first color channel 905, 906 may be modulated by the same modulationfunction, or alternatively, may be modulated using a separate set ofmodulation functions based on a different set of frequencies orcoefficients according to the image sensor configuration. In thisexample, the modulation functions for the first color channel 917 areidentical, as well as the modulation functions for the second colorchannel 918. FIG. 10 also includes two instances of a circle with a plussign (+) in the middle indicating a function of summation of themodulated components from the first color channel, and summation of themodulated components of the second color channel. As a result of a firstsummation 1001 of first channel baseband chrominance components 905,906, a first channel chrominance carrier frequency 1005 may begenerated. Similarly, as a result of a second summation 1002 of secondchannel baseband chrominance components 907, 908, a second channelchrominance carrier frequency 1006 may be generated. The basebandsignals may be expressed with the following equation:

$\begin{matrix}{{{\left\lbrack \underset{\_}{n} \right\rbrack} = {\sum\limits_{\underset{\_}{\lambda} \in \underset{\;}{{\hat{\Lambda}}_{M}^{*}{{\backslash(}{{0,0})}}}}^{\;}{{\left\lbrack \underset{\_}{n} \right\rbrack}{m_{c_{1}}^{\underset{\_}{\lambda}}\left\lbrack \underset{\_}{n} \right\rbrack}}}},{i = 1},2} & (7)\end{matrix}$

Where:

-   -   [n]: the modulated baseband chrominance signal for each color        channel of the chrominance components,    -   [n]: the baseband chrominance signal for each color channel of        the chrominance components,    -   m_(C) _(i) ^(λ) [n]: the modulation function representative of        the period of the CFA pattern.

FIG. 11 illustrates the two chrominance carrier frequencies for thefirst color channel 1005 and the second color channel 1006 describedabove, as well as a modulation function 1007 for the first color channelchrominance component and a modulation function 1008 for the secondcolor channel chrominance component. The first color channel chrominancecomponent 1005 may be modulated to create the full first channelchrominance component 1101. Similarly, the second color channelchrominance component 1006 may be modulated to create the full secondchannel chrominance component 1102.

FIG. 12 schematically illustrates an example extraction process 1200that extracts the luminance component 1225 from the single planespectral image 500 produced by the 3×3 Bayer pattern 200 illustrated inFIG. 2. In a first part 1202 of the extraction process 1200, a firstchrominance component 1210 is extracted from the single plane spectralimage 500. In a second part 1204 of the extraction process 1200, asecond chrominance component 1212 is extracted from the single planespectral image 500.

Block 1206 includes a baseband luminance component 1225 for thefull-channel image which may be estimated using the following equation:

l[n]=y[n]−

[n]−

[n].  (8)

Where:

-   -   [n]: the modulated baseband chrominance signal 1210 for the        first color channel of the chrominance components,    -   [n]: the baseband chrominance signal 1210 for the second color        channel of the chrominance components,    -   l[n]: the estimated baseband luminance component 1225.

FIG. 13A illustrates a flowchart of an example of a process 1300 forconverting image data generated by an image sensor into a second image.In some aspects, the image data may comprise any of the images 100, 200,or 300 discussed above. In some aspects, the image data may be anysingle plane image, with any configuration of image sensor elements andoverlying color filters. In some aspects, the image data may be just aportion of a complete image, such as a portion of images 100, 200, or300 discussed above.

As discussed above, the Bayer pattern is no longer the dominant colorfilter array (CFA) pattern in the sensor industry. Multiple color filterarray (CFA) patterns have gained popularity, including 1) color filterarrangements e.g. white pixel sensors, Lucas, PanChromatic, etc.; (2)color filter size based, e.g. configurations including a color filterthat is 2× the pixel size, configurations including color filters thatare 1.5× pixel size, etc; and (3) exposure based high dynamic range(HDR) sensors. Process 1300 provides a hardware-friendly universaldemosaic process that can demosaic data obtained from virtually anycolor filter array pattern.

Given an arbitrary CFA pattern, process 1300 may first determine aspectrum of the CFA image. The CFA spectrum demonstrates that mosaickingoperation is, essentially a frequency modulation operation. In someaspects, a luminance component of the image resides at baseband whilechrominance components of the image are modulated at high frequencies.After the CFA spectrum is derived, process 1300 may derive modulatingcarrier frequencies and modulating coefficients that may characterize aforward mosaicking operation. Given the modulating carrier frequenciesand coefficients, process 1300 may then derive one or more ofspatial-domain directional filters, spatial-domain modulation functions,and spatial-domain demodulation functions for performing a demosaicoperation.

In some aspects, process 1300 may be implemented by instructions thatconfigure an electronic hardware processor to perform one or more of thefunctions described below. For example, in some aspects, process 1300may be implemented by the device 1600, discussed below with respect toFIG. 14. Note that while process 1300 is described below as a series ofblocks in a particular order, one of skill in the art would recognizethat in some aspects, one or more of the blocks describes below may beomitted, and/or the relative order of execution of two or more of theblocks may be different than that described below.

Block 1305 receives information indicating a configuration of sensorelements of an image sensor and a configuration of filters for thesensor elements. For example, the information received in block 1305 mayindicate the image sensor configuration is any one of configurations100, 200, 300 discussed above. The image sensor configuration mayalternatively be any other sensor configuration. In someimplementations, the image sensor configuration may comprise an array ofsensor elements, each sensor element having a surface for receivingradiation, and each sensor element being configured to generate theimage data based on radiation that is incident on the sensor element.The image sensor configuration may include a CFA pattern that includesan array of filter elements disposed adjacent to the array of sensorelements to filter radiation propagating towards sensor elements in thearray of sensor elements.

In some aspects, the image sensor configuration may be dynamicallyderived in block 1305. In some embodiments, the image sensorconfiguration may be determined using information defining the CFApattern (e.g., arrangement of the CFA, periodicity of a filter elementin a repeated pattern of the CFA, a length dimension of a filterelement, a width dimension of a filter element) corresponding to thearray of sensor elements. In one exemplary embodiment, determining animage sensor configuration may include a processor configured to receiveinformation from which a hardware configuration of the image sensor(including the CFA) is determined. In some examples, a processor mayreceive information indicative of an image sensor hardware configurationand determine the hardware information by accessing a look-up table orother stored information using the received information. In someexemplary embodiments, the image sensor may send configuration data tothe processor. In still another exemplary embodiments, one or moreparameters defining the image sensor configuration may be hard coded orpredetermined and dynamically read (or accessed) from a storage locationby an electronic processor performing process 1300.

Block 1310 generates a modulation function based on an image sensorconfiguration, which includes at least the information indicating theconfiguration of sensor elements of the image sensor and theconfiguration of filters for the sensor elements. The variety of exampleimage sensor configurations discussed above may allow generation of aset of sub-lattice parameters unique to a particular one image sensorconfiguration. The sub-lattice parameters of a given image sensorconfiguration are a set of properties of the image sensor, and one ormore of the set of properties may be used to generate an associatedmodulation function for the image sensor configuration. In some aspects,the sub-lattice parameters may be used to generate one or moremodulation frequencies and/or a set of modulation coefficients. One ormore of these generated components may be used to demosaic raw imagedata output by the particular image sensor. The sub-lattice parametersmay be made up of one or more of the following components:

et the symbol W represent the spectral components of the CFA pattern.

This may be a range of wavelengths the sensor element is exposed to, andcan be directly associated with the filter element or the plurality offilter elements that overlay each sensor element in a period of a CFApattern.

Let ({B_(S)}_(SεΨ)) represent coset vectors associated with a period ofa CFA pattern. For example, the traditional 2×2 Bayer pattern 100 ofFIG. 1 has 4 unique addresses (e.g., four sensor elements) in a CFAperiod where each address characterized as a location having ahorizontal property and a vertical property. For example, the 2×2pattern may be associated with a two-dimensional Cartesian coordinatesystem where the bottom left sensor element 103 of the 2×2 patterncorresponds with the origin, or address (0, 0). The bottom left sensorelement 103 being associated with a green filter element, the cosetvector at that particular sensor element would provide B_(G)={(0,0)}.The sensor element 102 directly above the bottom left image sensor 103,being exposed to red wavelength would then correspond to address (0, 1),resulting in coset vector B_(R)={(0,1)}. The sensor element 104 directlyto the right of the bottom left sensor element 103, being exposed to ablue wavelength would correspond to address (1, 0), resulting in cosetvector B_(B)={(1,0)} and the sensor element 102 directly above it wouldcorrespond to address (1, 1), providing B_(G)={(1,1)}. Due to the sensorelement 102 also being associated with a green filter element, the cosetvectors for the green spectral range would provide B_(G)={(0,0), (1,1)}.

A lattice matrix, or matrix generator, represented by (M). In someaspects, the matrix generator (M) may be a diagonal representation oftwo addresses, n and m, resulting in a 2×2 matrix. The first element ofthe matrix, being the number in the top left, is a number of sensorelements in one period of a CFA pattern in the x-direction of theperiod. For example, with a 2×2 Bayer pattern, such as pattern 100 shownin FIG. 1, the number of sensor elements in the x-direction is 2. Thesecond element of the matrix, being the number in the bottom right, is anumber of sensor elements in one period of the CFA pattern in they-direction of the CFA pattern. Using the 2×2 Bayer pattern 100 in FIG.1, the number of sensor elements in the y-direction is 2. The other twovalues in the matrix M are constant, and equal to zero (0).

Example values for the sub-lattice parameters for example image sensorconfigurations are as follows:

2 × 2 Bayer CFA pattern 100 3 × 3 Sensor Element Array 200 Ψ R,G,BR,G,B,C,Y,W B_(R) = {(2,0)}, B_(R) = {(0,1)}, B_(B) = {(0,2)},{B_(S)}_(S ε) _(Ψ) B_(B) = {(1,0)}, B_(W) = {(1,1)}, B_(G) = {(0,0),(1,1)}. B_(G) = {(0,0), (2,2)}, B_(M) = {(1,0), (2, 1)}, B_(C) = {(0,1),(1,2)}. M $\quad\begin{bmatrix}2 & 0 \\0 & 2\end{bmatrix}$ $\quad\begin{bmatrix}3 & 0 \\0 & 3\end{bmatrix}$

The Ψ component represents the spectral range of exposure to sensorelements in a period of filter elements. For example, in a traditionalBayer pattern, the spectral range is Red, Green, and Blue, and thus thespectral components Ψ={R,G,B}. In another example, where the colorfilter elements are 1.5 times the sensor element size, and use thetraditional Bayer spectral range (RGB), the spectral componentsW={R,G,B,C,Y,W}. Since the sensor elements in the “1.5” configurationmay be exposed to as many as four filter elements, there is a broaderwavelength exposure to a sensor element as compared to a sensor elementin a configuration where it is shielded by a single filter element of asingle color.

In this example, a sensor element may be exposed to a combination ofgreen and red wavelengths resulting in a light spectrum that can includeyellow (570-590 nm wavelength). Using the same example, a sensor elementmay be exposed to a combination of green and blue wavelengths resultingin a light spectrum that includes the color cyan (490-520 nmwavelength). The 2×2 filter matrix of this example may also be arrangedso that another sensor element is masked 25% by a filter element thatpasses a range of red light, 50% by filter elements that pass a range ofgreen light, and 25% by a filter element that passes a range of bluelight, thereby exposing that sensor element to a spectrum of light thatis broader than the spectrum exposed to the remaining sensors. Theresulting array has an effective sensor composition of 11% R, W, and B,respectively and 22% G and C, respectively, and the spectral componentscan be written as W={R,G,B,C,Y,W}. {B_(S)}_(SεΨ) represents mutuallyexclusive sets of coset vectors associated with the spatial samplinglocations of various filter elements in the period of filter elements.Lattice matrix M may be determined based on the number of filterelements in the period of filter elements and the number of pixels inthe same. M may also be referred to herein as a generator matrix.

Further to block 1310, a frequency domain analysis can be done on anarbitrary CFA pattern using the Fourier transform of the particular CFApattern. The Fourier transform of a CFA pattern is given by:

$\begin{matrix}{{Y\left( \underset{\_}{f} \right)} = {\frac{1}{{\det (M)}}{\sum\limits_{S \in \Psi}^{\;}\; {\sum\limits_{\underset{\_}{\lambda} \in {\hat{\Lambda}}_{M}}{\sum\limits_{\underset{\_}{b} \in B_{S}}\; {e^{{- j}\; 2\; \pi {\underset{\_}{b}}^{T}\underset{\_}{\lambda}}{S\left( {\underset{\_}{f}\; - \underset{\_}{\lambda}} \right)}}}}}}} & (9)\end{matrix}$

Where:

-   -   Y(f): the frequency transform of any period of an image sensor,    -   M: the lattice matrix representative of a period of the image        sensor,    -   S: spectral component that is currently being analyzed, where S        is an element of Ψ, where Ψ includes all of the spectral        elements of one period of the CFA pattern,    -   S(f−λ): Fourier transform of the spectral component S in one        period of the CFA pattern,    -   b ^(T): transposed cosite vector associated with a spectral        component in one period of the CFA pattern,    -   SεΨ: a particular spectral component “S” present in a period of        the CFA pattern,    -   λε{circumflex over (Λ)}_(M): {circumflex over (Λ)}_(M) is a set        of all modulation carrier frequencies of a given CFA period, λ        represents a particular carrier frequency of that set,    -   bεB_(S): B_(S) is a set of all cosite vectors associated with a        spectral component in one period, b represents a particular        cosite vector of that set.

In equation (9) above, {circumflex over (Λ)}_(M) may be referred to asthe dual lattice associated with a corresponding lattice matrix M, alsoknown as a “generator matrix,” and is given by:

Λ ^ M = M - T  m _ ⋂ ( - 1 2 , 1 2 ] 2 ,  where   m _ ∈ 2 . ( 10 )

Where:

-   -   {circumflex over (Λ)}_(M): set of all modulation carrier        frequencies of a given CFA period,    -   M^(−T): inverse transpose of the lattice matrix M,    -   m: a point in the spectral domain, an integer on a 2d grid        (x,y), the 2d grid being the spectral domain of a Fourier        transform, and    -   λ—a particular modulation frequency in the set of modulation        frequencies.

Rearranging the terms in equation (9) provides the following:

$\begin{matrix}{{Y\left( \underset{\_}{f} \right)} = {{\frac{1}{{\det (M)}}{\sum\limits_{S \in \Psi}^{\;}\; {\sum\limits_{\underset{\_}{b} \in B_{S}}{S\left( \underset{\_}{f} \right)}}}} + {\sum\limits_{\underset{\_}{\lambda} \in \underset{\;}{{\hat{\Lambda}}_{M}{{\backslash(}{{0,0})}}}}^{\;}\left\{ {\frac{1}{{\det (M)}}{\sum\limits_{S \in \Psi}^{\;}\; {\sum\limits_{\underset{\_}{b} \in B_{S}}{e^{{- j}\; 2\; \pi {\underset{\_}{b}}^{T}\underset{\_}{\lambda}}{S\left( {\underset{\_}{f} - \underset{\_}{\lambda}} \right)}}}}} \right\}}}} & (11)\end{matrix}$

Where:

S(f): Fourier transform for the spectral component S in one period ofthe CFA pattern.

The first term in equation (11),

$\left( {{i.e.},{\frac{1}{{\det (M)}}{\sum\limits_{S \in \Psi}^{\;}\; {\sum\limits_{\underset{\_}{b} \in B_{S}}{S\left( \underset{\_}{f} \right)}}}}} \right)$

comprises the baseband luminance component and, since Σ_(SεΨ)Σ _(bεB)_(S) e^(−j2π) ^(T) ^(λ)=0 for λε{circumflex over (Λ)}_(M)\(0,0), thesecond term in equation (11),

$\left( {\sum\limits_{\underset{\_}{\lambda} \in \underset{\;}{{\hat{\Lambda}}_{M}{{\backslash(}{{0,0})}}}}^{\;}\left\{ {\frac{1}{{\det (M)}}{\sum\limits_{S \in \Psi}^{\;}\; {\sum\limits_{\underset{\_}{b} \in B_{S}}{e^{{- j}\; 2\; \pi {\underset{\_}{b}}^{T}\underset{\_}{\lambda}}{S\left( {\underset{\_}{f} - \underset{\_}{\lambda}} \right)}}}}} \right\}} \right)$

represents the high-pass chrominance components modulated to frequenciesλε{circumflex over (Λ)}_(M) \(0,0). Each chrominance component comprisesa complex weighted sum of all spectral components present in one periodof the CFA, with the complex weights adding up to zero. Denoted by L andC _(λ) , the luminance and modulated chrominance components,respectively, can be written as:

Y( f )=L( f )+Σ _(λε{circumflex over (Λ)}) _(M) _(\(0,0)) C _(λ) ( f−λ).  (12)

Where:

$\begin{matrix}{{{C_{\underset{\_}{\lambda}}\left( {\underset{\_}{f} - \underset{\_}{\lambda}} \right)} = {\frac{1}{{\det (M)}}{\sum\limits_{S \in \Psi}^{\;}\; {\sum\limits_{\underset{\_}{b} \in B_{S}}{e^{{- j}\; 2\; \pi {\underset{\_}{b}}^{T}\underset{\_}{\lambda}}{S\left( {\underset{\_}{f} - \underset{\_}{\lambda}} \right)}}}}}},\mspace{14mu} {and}} & (13) \\{{L\left( \underset{\_}{f} \right)} = {\frac{1}{{\det (M)}}{\sum\limits_{S \in \Psi}^{\;}\; {\sum\limits_{\underset{\_}{b} \in B_{S}}{{S\left( \underset{\_}{f} \right)}.}}}}} & (14)\end{matrix}$

Thus, (12) allows for an arbitrary CFA pattern to be decomposed intobaseband luminance and modulated chrominance components. The Fouriertransform in equation (12) may be simplified as follows:

$\begin{matrix}{{Y\left( \underset{\_}{f} \right)} = {{L\left( \underset{\_}{f} \right)} + {\sum\limits_{\underset{\_}{\lambda} \in \underset{\;}{{\hat{\Lambda}}_{M}{{\backslash(}{{0,0})}}}}^{\;}{\left\{ {{S_{\underset{\_}{\lambda}}{C_{1}\left( {\underset{\_}{f} - \underset{\_}{\lambda}} \right)}} + {t_{\underset{\_}{\lambda}}{C_{2}\left( {\underset{\_}{f} - \underset{\_}{\lambda}} \right)}}} \right\}.}}}} & (15)\end{matrix}$

A distinction between equation (12) and equation (15) is that in thelatter there are two unique chrominance components, C₁ and C₂, and eachchrominance component is a real-weighted sum of all spectral components,SεΨ, present in a period of the CFA. The modulation coefficients s _(λ)and t _(λ) are, in general, complex, with s_(−λ) =s _(λ) * and t_(−λ) =t_(λ) * whenever λ, −λε{circumflex over (Λ)}_(M) \(0,0).

For a given periodic CFA pattern, the lattice generator matrix M, theset of spectral filters in one period of the CFA (Ψ), and the sets ofoffset vectors associated with spectral filters {B_(S)}_(SεΨ) can beinferred as explained above. For instance, the values of Ψ, B_(S), and Mfor the two example CFA patterns 100 and 200 shown in FIG. 1 and FIG. 2are defined above. Substituting the values of Ψ, B_(S), and M inequation (9) and equation (10), the modulation carrier frequenciesλε{circumflex over (Λ)}_(M)\(0,0), the modulation coefficients s _(λ)and t _(λ) , and the inherent RGB to LC₁C₂ 3×3 transformation for agiven CFA pattern may be determined. Taking the inverse Fouriertransform of equation (15) enables expression of the CFA pattern y[n] interms of the luminance and chrominance components for arbitrary CFApatterns:

y[n]=l[n]+m _(C) ₁ [n]C ₁ [n]+m _(C) ₂ [n]C ₂ [n]  (16)

Where:

l[n]: Luminance component at point n,

-   -   m_(C) ₁ [n]: spatial domain modulation function for first color        channel chrominance component,    -   c₁[n]: spatial domain of a first color channel chrominance        component,    -   m_(C) ₂ [n]: spatial domain modulation function for second color        channel chrominance component,    -   c₂[n]: spatial domain of a second color channel chrominance        component.

In equation (16), m_(C) ₁ [n] and m_(C) ₂ [n] represent spatial-domainfunctions that modulate the chrominance signals to high-frequencycarrier waves. The modulation function for chrominance channels C1 andC2 can be given by:

m c 1  [ n _ ] = - 1  [ ∑ λ _ ∈ Λ ^ M  \(  0 , 0 )  { s λ _  δ  (f _ - λ _ ) } ] ,  and ( 17  a ) m c 2  [ n _ ] = - 1  [ ∑ λ _ ∈ Λ ^M  \(  0 , 0 )  { t λ _  δ  ( f _ - λ _ ) } ] ( 17  b )

Where:

-   -   ⁻¹: inverse Fourier transform,    -   s _(λ) : modulation coefficient for first color channel        chrominance component,    -   t _(λ) : modulation coefficient for second color channel        chrominance component,    -   δ(f−λ): delta represents a Dirac delta function, meaning the        delta function is equal to zero when f−λ is not equal to zero.        Equal to infinity when f=λ.

Defining the set {circumflex over (Λ)}_(M)*={λε{circumflex over(Λ)}_(M)|λ₁=0,λ₂>0}∪{λε{circumflex over (Λ)}_(M)|λ₁>0}, the equation(17a) and equation (17b) may be re-written as:

m c 1  [ n _ ] = - 1  [ ∑ λ _ ∈ Λ ^ M *  \(  0 , 0 )  s λ _  δ  (f _ - λ _ ) + s  _ λ _  δ  ( f _ - λ _ ) ] ,  and ( 18  a ) m c 2 [ n _ ] = - 1  [ ∑ λ _ ∈ Λ ^ M *  \(  0 , 0 )  t λ _  δ  ( f _ - λ_ ) + t  _ λ _  δ  ( f _ + λ _ ) ] ( 18  b )

Where:

-   -   s_(−λ) : modulation coefficient for the first color channel        chrominance component at the negative of the 2d vector        represented by lambda,    -   t_(−λ) : modulation coefficient for the second color channel        chrominance component at the negative of the 2d vector        represented by lambda.

As noted above, s_(−λ) =s _(λ) * when −λε{circumflex over (Λ)}_(M)\(0,0) and equals zero otherwise. Computing the inverse Fouriertransform of equation (18a) and equation (18b) provides thespatial-domain frequency-modulation function for the chrominance channelC1 and C2:

$\begin{matrix}{{{m_{c_{1}}\left\lbrack \underset{\_}{n} \right\rbrack} = {\sum\limits_{\underset{\_}{\lambda} \in \underset{\;}{{\hat{\Lambda}}_{M}^{*}{{\backslash(}{{0,0})}}}}^{\;}{m_{c_{1}}^{\underset{\_}{\lambda}}\left\lbrack \underset{\_}{n} \right\rbrack}}},\mspace{14mu} {and}} & \left( {19a} \right) \\{{{m_{c_{2}}\left\lbrack \underset{\_}{n} \right\rbrack} = {\sum\limits_{\underset{\_}{\lambda} \in \underset{\;}{{\hat{\Lambda}}_{M}^{*}{{\backslash(}{{0,0})}}}}^{\;}\; {m_{c_{2}}^{\underset{\_}{\lambda}}\left\lbrack \underset{\_}{n} \right\rbrack}}}{{Where},}} & \left( {19b} \right) \\{{m_{c_{1}}^{\underset{\_}{\lambda}}\left\lbrack \underset{\_}{n} \right\rbrack} = \left\{ {\begin{matrix}{{\sum\limits_{\underset{\_}{\lambda} \in \underset{\;}{{\hat{\Lambda}}_{M}^{*}{{\backslash(}{{0,0})}}}}^{\;}{s_{\underset{\_}{\lambda}}e^{j\; 2\; \pi {\underset{\_}{\lambda}}^{T}\underset{\_}{n}}}},{{- \underset{\_}{\lambda}} \notin {\hat{\Lambda}}_{M}}} \\{{\sum\limits_{\underset{\_}{\lambda} \in \underset{\;}{{\hat{\Lambda}}_{M}^{*}{{\backslash(}{{0,0})}}}}^{\;}{{s_{\underset{\_}{\lambda}}}{\cos \left( {{2\; \pi {\underset{\_}{\lambda}}^{T}\underset{\_}{n}} + \phi_{s_{\underset{\_}{\lambda}}}} \right)}}},{{- \underset{\_}{\lambda}} \in {\hat{\Lambda}}_{M}}}\end{matrix},{and}} \right.} & \left( {20a} \right) \\{{m_{c_{2}}^{\underset{\_}{\lambda}}\left\lbrack \underset{\_}{n} \right\rbrack} = \left\{ \begin{matrix}{{\sum\limits_{\underset{\_}{\lambda} \in \underset{\;}{{\hat{\Lambda}}_{M}^{*}{{\backslash(}{{0,0})}}}}^{\;}{t_{\underset{\_}{\lambda}}e^{j\; 2\; \pi {\underset{\_}{\lambda}}^{T}\underset{\_}{n}}}},{{- \underset{\_}{\lambda}} \notin {\hat{\Lambda}}_{M}}} \\{{\sum\limits_{\underset{\_}{\lambda} \in \underset{\;}{{\hat{\Lambda}}_{M}^{*}{{\backslash(}{{0,0})}}}}^{\;}{{t_{\underset{\_}{\lambda}}}{\cos \left( {{2\; \pi {\underset{\_}{\lambda}}^{T}\underset{\_}{n}} + \phi_{t_{\underset{\_}{\lambda}}}} \right)}}},{{- \underset{\_}{\lambda}} \in {\hat{\Lambda}}_{M}}}\end{matrix} \right.} & \left( {20b} \right)\end{matrix}$

In equation (20a), |s _(λ) | and φ _(λ) respectively define theamplitude and the phase of the complex modulation coefficient s _(λ) ,where:

|S _(λ) |=√{square root over (Re{s _(λ) }² +Im{s _(λ) }²)},φs _(λ)=arctan 2(Im{s _(λ) },Re{s _(λ) }).   (21)

Where:

-   -   Re{s _(λ) } refers to a real number portion of the modulation        coefficients at a set of x y coordinates within the spatial        domain of a given CFA signal in an image, and    -   Im{s _(λ) } refers to an imaginary portion of the modulation        coefficients at the set of x y coordinates within the spatial        domain of the image.

In equation (20b), |t _(λ) | and φ _(λ) respectively define theamplitude and the phase of the complex modulation coefficient s _(λ) ,where:

|t _(λ) |=√{square root over (Re{t _(λ) }² +Im{t _(λ) }²)},φt _(λ)=arctan 2(Im{t _(λ) },Re{t _(λ) }).  (22)

Where:

-   -   Re{t _(λ) } refers to a real number portion of the modulation        coefficients at a set of x y coordinates within the spatial        domain of a given CFA signal in an image, and    -   Im{t _(λ) } refers to an imaginary portion of the modulation        coefficients at the set of x y coordinates within the spatial        domain of the image.

Further to block 1310, as discussed above, the modulation function isdetermined based on a set of modulation frequencies and the set ofmodulation coefficients derived as discussed above. The modulationfrequencies may be described as {circumflex over(Λ)}_(M)=Mm∩(Λε{circumflex over (Λ)}_(M)\(0,0)]², where {circumflex over(Λ)}_(M) is a set of modulation frequencies of a given CFA period, mε

² where

² is a two dimensional integer lattice of a spatial domain of the image,M is the lattice matrix determined based on the sensor configurationdescribed above, and Λ is equivalent to the integer lattice

².

In block 1315, image data is demodulated based on the generatedmodulation function to determine chrominance and luminance components ofthe image data. In some aspects, block 1315 may perform the functionsdescribed below with respect to process 1315 of FIG. 13B. In someaspects, the image data that is demodulated may comprise an image, forexample, an image of a scene captured by the image sensor. In someaspects, the image data may comprise only a portion of an image capturedby the image sensor. In some aspects, the image data comprises a singleplane image.

In block 1320, a triple plane image is generated based on the determinedchrominance and luminance components. As disclosed above, the singleplane CFA image comprises sections of luminance and chrominancecomponents in a spatial frequency domain. For example, FIG. 7illustrates generation of an image based on extracted luminancecomponent and baseband signals for each chrominance component. In someaspects of block 1320, an image other than a triple plan image may begenerated. For example, in some aspects, a single plane, or double planeimage may be generated instead of a triple plane image.

FIG. 13B is a flowchart of one example of a process for demodulating animage. In some aspects, process 1315 of FIG. 13B may be performed by theprocessor 1404 of FIG. 14A or 14B. In some aspects, process 1315 may beperformed by the universal demosaic 1432 of FIG. 14B.

In block 1355, a set of configuration parameters are generated based ona derived modulation function. In one example embodiment, the generatedconfiguration parameters may include a set of high pass frequencyfilters configured to extract the set of chrominance components from aCFA image. In some aspects, the high pass filters may be modulated basedon the configuration of the image sensor to perform the extraction. Theconfiguration parameters may also include a set of edge detectingfilters configured to determine an energy level of the image data in atleast one or more of a horizontal direction, a vertical direction, and adiagonal direction. The edge detecting filters may also be configured todetect an energy level indicative of an intensity difference ofradiation that is incident on neighboring sensor elements. Thus, theedge detection filters may be configured to identify points in a digitalimage at which the image brightness changes sharply, or has adiscontinuity.

In block 1360, chrominance components from the image data are extractedbased on the generated set of configuration parameters. As disclosedabove, the image data comprises luminance and chrominance components ina spatial frequency domain. For example, FIG. 8 illustrates a method forfiltering a single plane spectral image 500 to extract the chrominancecomponents 501, 502 using a filter set (for example, filters 800, 801,802, 803).

High pass filters may be used to extract modulated chrominancecomponents from the image data. In one exemplary embodiment, a pair ofhigh-pass filters

${{h_{1}^{\underset{\_}{\lambda}}\left\lbrack \underset{\_}{n} \right\rbrack}\mspace{14mu} {and}\mspace{14mu} {h_{2}^{\underset{\_}{\lambda}}\left\lbrack \underset{\_}{n} \right\rbrack}}\mspace{14mu}$

are designed based on the derived modulation function to extractmodulated chrominance components, where

${h_{1}^{\underset{\_}{\lambda}}\left\lbrack \underset{\_}{n} \right\rbrack}\mspace{20mu}$

may extract the C1 401, 501, 601 chrominance components resulting in afiltered product described as:

$\begin{matrix}{{{c_{1m}^{\underset{\_}{\lambda}}\left\lbrack \underset{\_}{n} \right\rbrack}\overset{\Delta}{=}{{m_{c_{1}}^{\underset{\_}{\lambda}}\left\lbrack \underset{\_}{n} \right\rbrack}{c_{1}\left\lbrack \underset{\_}{n} \right\rbrack}}},} & \left( {23\; a} \right)\end{matrix}$

Where

-   -   c_(1m) λ[n] represents the extracted C1 chrominance component,    -   m_(C) ₁ ^(λ) [n] represents the modulation function, and    -   c₁[n] represents the C1 chrominance component before extraction        using the filter for each λε{circumflex over (Λ)}*_(M)\(0,0)        from a given CFA pattern y[n].

The extracted C2 chrominance component may be described as:

c _(2m) ^(λ) [n]

m _(C) ₂ ^(λ) [n]c ₂ [n]  (23b)

Where:

-   -   c_(2m) ^(λ) [n] represents the extracted chrominance component,    -   m_(c) ₂ ^(λ) [n] represents the modulation function for the C2        component, and    -   c₂[n] represents the C2 chrominance component before extraction        using the filter for each λε{circumflex over (Λ)}*_(M)\(0,0)        from a given CFA pattern y[n].

The filtering equations are given by:

[ n]=Σ _(m) y[n]h _(i) ^(λ) [n−m],  (24)

Where

-   -   i=1,2 and is representative of the set of chrominance        components, and    -   [y[n] is the particular CFA pattern being analyzed, in this        case, the CFA pattern of the image data.

The edge detection filters may be generated in a similar manner, byusing the derived modulation function or by using a known set of edgedetectors. The edge detection filters may similarly be generated usingthe modulation function for the image data or by using a known set ofedge detectors.

In block 1365, the extracted chrominance components are demodulated intoa set of baseband chrominance components. As disclosed above, theextracted chrominance components 808, 809, 810, 811 can be demodulatedusing the following equation:

$\begin{matrix}{= \left\{ {\begin{matrix}{\frac{\left\lbrack \underset{\_}{n} \right\rbrack}{m_{c_{1}}^{\underset{\_}{\lambda}}\left\lbrack \underset{\_}{n} \right\rbrack},} & {{m_{c_{1}}^{\underset{\_}{\lambda}}\lbrack n\rbrack} \neq 0} \\0 & {{m_{c_{1}}^{\underset{\_}{\lambda}}\left\lbrack \underset{\_}{n} \right\rbrack} = 0}\end{matrix},{i = 1},2.} \right.} & (25)\end{matrix}$

For example, FIG. 9 illustrates the demodulation of the chrominancecomponents extracted using the high pass filtering derived from themodulation function into a set of baseband chrominance components.

In block 1370, the baseband chrominance components are modulated totheir respective carrier frequencies. As disclosed above, the basebandchrominance signals can be multiplied with the modulation functions ofluminance and chrominance components in a spatial frequency domain. Forexample, FIG. 11 illustrates one aspect of block 1370.

In block 1375, a luminance component is extracted from the image databased on the determined carrier frequencies. In some aspects, themodulated chrominance components are subtracted from the image data todetermine the luminance component. As disclosed above, the single planeCFA image comprises sections of luminance and chrominance components ina spatial frequency domain. For example, FIG. 12 and the correspondingdiscussion illustrate one aspect of block 1375. In some aspects, theluminance component may be obtained by subtracting all chrominancecomponents from the image data.

FIG. 14A shows an exemplary functional block diagram of a wirelessdevice 1402 a that may implement one or more of the disclosedembodiments. The wireless device 1402 a may include a processor 1404which controls operation of the wireless device 1402 a. The processor1404 may also be referred to as a central processing unit (CPU). Memory1406 a, which may include both read-only memory (ROM) and random accessmemory (RAM), may provide instructions and data to the processor 1404. Aportion of the memory 1406 a may also include non-volatile random accessmemory (NVRAM). The processor 1404 typically performs logical andarithmetic operations based on program instructions stored within thememory 1406 a. The instructions in the memory 1406 a may be executableto implement the methods described herein.

The processor 1404 may comprise or be a component of a processing systemimplemented with one or more processors. The one or more processors maybe implemented with any combination of general-purpose microprocessors,microcontrollers, digital signal processors (DSPs), field programmablegate array (FPGAs), programmable logic devices (PLDs), controllers,state machines, gated logic, discrete hardware components, dedicatedhardware finite state machines, or any other suitable entities that canperform calculations or other manipulations of information.

The processing system may also include machine-readable media forstoring software. Software shall be construed broadly to mean any typeof instructions, whether referred to as software, firmware, middleware,microcode, hardware description language, or otherwise. Instructions mayinclude code (e.g., in source code format, binary code format,executable code format, or any other suitable format of code). Theinstructions, when executed by the one or more processors, cause theprocessing system to perform the various functions described herein.

The wireless device 1402 a may also include a housing 1408 that mayinclude a transmitter 1410 and/or a receiver 1412 to allow transmissionand reception of data between the wireless device 1402 a and a remotelocation. The transmitter 1410 and receiver 1412 may be combined into atransceiver 1414. An antenna 1416 may be attached to the housing 1408and electrically coupled to the transceiver 1414. An image sensor 1430may capture images and make image data available to the processor 1404.In some aspects, the image sensor 1430 may be configured to capture anyone or more of the images 100, 200, or 300 discussed herein. Thewireless device 1402 a may also include (not shown) multipletransmitters, multiple receivers, multiple transceivers, and/or multipleantennas.

The wireless device 1402 a may also include a signal detector 1418 thatmay be used in an effort to detect and quantify the level of signalsreceived by the transceiver 1414. The signal detector 1418 may detectsuch signals as total energy, energy per subcarrier per symbol, powerspectral density and other signals. The wireless device 1402 a may alsoinclude a digital signal processor (DSP) 1420 for use in processingsignals. The DSP 1420 may be configured to generate a packet fortransmission. In some aspects, the packet may comprise a physical layerdata unit (PPDU).

The wireless device 1402 a may further comprise a user interface 1422 insome aspects. The user interface 1422 may comprise a keypad, amicrophone, a speaker, and/or a display. The user interface 1422 mayinclude any element or component that conveys information to a user ofthe wireless device 1402 a and/or receives input from the user.

The various components of the wireless device 1402 a may be coupledtogether by a bus system 1426. The bus system 1426 may include a databus, for example, as well as a power bus, a control signal bus, and astatus signal bus in addition to the data bus. Those of skill in the artwill appreciate the components of the wireless device 1402 a may becoupled together or accept or provide inputs to each other using someother mechanism.

Although a number of separate components are illustrated in FIG. 15,those of skill in the art will recognize that one or more of thecomponents may be combined or commonly implemented. For example, theprocessor 1404 may be used to implement not only the functionalitydescribed above with respect to the processor 1404, but also toimplement the functionality described above with respect to the signaldetector 1418 and/or the DSP 1420. Further, each of the componentsillustrated in FIG. 14 may be implemented using a plurality of separateelements.

The wireless device 1402 a may be used to transmit and/or receivecommunications. Certain aspects contemplate signal detector 1418 beingused by software running on memory 1406 a and processor 1404 to detectthe presence of a transmitter or receiver.

FIG. 14B shows an exemplary functional block diagram of a wirelessdevice 1402 b that may implement one or more of the disclosedembodiments. The wireless device 1402 b may include components similarto those shown above with respect to FIG. 14B. For example, the device1402 b may include a processor 1404 which controls operation of thewireless device 1402 b. The processor 1404 may also be referred to as acentral processing unit (CPU). Memory 1406 b, which may include bothread-only memory (ROM) and random access memory (RAM), may provideinstructions and data to the processor 1404. A portion of the memory1406 b may also include non-volatile random access memory (NVRAM). Theprocessor 1404 typically performs logical and arithmetic operationsbased on program instructions stored within the memory 1406 b. Theinstructions in the memory 1406 b may be executable to implement themethods described herein. In some aspects, the instructions stored inthe memory 1406 b may differ from the instructions stored in the memory1406 a of FIG. 14A. For example, as discussed above, in some aspects,the processor 1404 of FIG. 14A may be configured by instructions storedin the memory 1406 a to perform one or more of the methods disclosedherein. In the alternative, the processor 1404 in the device 1402 b mayperform the methods disclosed herein in concert with a universaldemosaic component 1432, discussed below.

The processor 1404 may comprise or be a component of a processing systemimplemented with one or more processors. The one or more processors maybe implemented with any combination of general-purpose microprocessors,microcontrollers, digital signal processors (DSPs), field programmablegate array (FPGAs), programmable logic devices (PLDs), controllers,state machines, gated logic, discrete hardware components, dedicatedhardware finite state machines, or any other suitable entities that canperform calculations or other manipulations of information.

The processing system may also include machine-readable media forstoring software. Software shall be construed broadly to mean any typeof instructions, whether referred to as software, firmware, middleware,microcode, hardware description language, or otherwise. Instructions mayinclude code (e.g., in source code format, binary code format,executable code format, or any other suitable format of code). Theinstructions, when executed by the one or more processors, cause theprocessing system to perform the various functions described herein.

The universal demosaic component 1432 may be configured to demosaic datareceived from the image sensor 1430. The universal demosaic 1432 mayreceive information defining a configuration of the image sensor fromone or more of the processor 1404 and/or the image sensor 1430. Theconfiguration data may include data indicating a configuration of imagesensor elements of the image sensor 1430, for example, as describedabove with respect to FIG. 1, 2 or 3, and information indicating aconfiguration of filters that filter light before it reaches the imagesensor elements. Based at least on the received image sensorconfiguration information, the universal demosaic may demosaic datagenerated by the image sensor 1430. The universal demosaic component maythen output data defining a triple plane image onto the data bus 1426.

The wireless device 1402 b may also include a housing 1408 that mayinclude a transmitter 1410 and/or a receiver 1412 to allow transmissionand reception of data between the wireless device 1402 b and a remotelocation. The transmitter 1410 and receiver 1412 may be combined into atransceiver 1414. An antenna 1416 may be attached to the housing 1408and electrically coupled to the transceiver 1414. An image sensor 1430may capture images and make image data available to the processor 1404.In some aspects, the image sensor 1430 may be configured to capture anyone or more of the images 100, 200, or 300 discussed herein. Thewireless device 1402 b may also include (not shown) multipletransmitters, multiple receivers, multiple transceivers, and/or multipleantennas.

The wireless device 1402 b may also include a signal detector 1418 thatmay be used in an effort to detect and quantify the level of signalsreceived by the transceiver 1414. The signal detector 1418 may detectsuch signals as total energy, energy per subcarrier per symbol, powerspectral density and other signals. The wireless device 1402 b may alsoinclude a digital signal processor (DSP) 1420 for use in processingsignals. The DSP 1420 may be configured to generate a packet fortransmission. In some aspects, the packet may comprise a physical layerdata unit (PPDU).

The wireless device 1402 b may further comprise a user interface 1422 insome aspects. The user interface 1422 may comprise a keypad, amicrophone, a speaker, and/or a display. The user interface 1422 mayinclude any element or component that conveys information to a user ofthe wireless device 1402 b and/or receives input from the user.

The various components of the wireless device 1402 b may be coupledtogether by a bus system 1426. The bus system 1426 may include a databus, for example, as well as a power bus, a control signal bus, and astatus signal bus in addition to the data bus. Those of skill in the artwill appreciate the components of the wireless device 1402 b may becoupled together or accept or provide inputs to each other using someother mechanism.

Although a number of separate components are illustrated in FIG. 15,those of skill in the art will recognize that one or more of thecomponents may be combined or commonly implemented. For example, theprocessor 1404 may be used to implement not only the functionalitydescribed above with respect to the processor 1404, but also toimplement the functionality described above with respect to the signaldetector 1418 and/or the DSP 1420. Further, each of the componentsillustrated in FIG. 14 may be implemented using a plurality of separateelements.

The wireless device 1402 b may be used to transmit and/or receivecommunications. Certain aspects contemplate signal detector 1418 beingused by software running on memory 1406 b and processor 1404 to detectthe presence of a transmitter or receiver.

FIG. 15 is a functional block diagram of an exemplary device 1500 thatmay implement one or more of the embodiments disclosed above. The device1500 includes an image sensor configuration determination circuit 1505.In an embodiment, the determination circuit 1505 may be configured toperform one or more of the functions discussed above with respect toblock 1305. In an embodiment, the determination circuit 1505 may includean electronic hardware processor, such as processor 1404 of FIG. 14A or14B. The determination circuit 1505 may also include one or more of aprocessor, signal generator, transceiver, decoder, or a combination ofhardware and/or software component(s), circuits, and/or module(s).

The device 1500 further includes a modulation function generationcircuit 1507. In an embodiment, the modulation function generationcircuit 1507 may be configured to perform one or more of the functionsdiscussed above with respect to block 1310. In an embodiment, themodulation function generation circuit 1507 may include an electronichardware processor, such as processor 1404 of FIG. 14A or 14B. In someaspects, the modulation function generation circuit 1507 may compriseone or more of a processor, signal generator, transceiver, decoder, or acombination of hardware and/or software component(s), circuits, and/ormodule(s). In some aspects, the modulation function generation circuit1507 may include the universal demosaic 1432 shown above in FIG. 14B.

The device 1500 further includes a parameter generation circuit 1510. Inan embodiment, the parameter generation circuit 1510 may be configuredto perform one or more of the functions discussed above with respect toblock 1355. In an embodiment, the parameter generation circuit 1510 mayinclude an electronic hardware processor, such as processor 1404 of FIG.14A or 14B. In some aspects, the parameter generation circuit 1510 maycomprise one or more of a processor, signal generator, transceiver,decoder, or a combination of hardware and/or software component(s),circuits, and/or module(s).

The device 1500 further includes a chrominance extraction circuit 1515.In an embodiment, the chrominance extraction circuit 1515 may beconfigured to perform one or more of the functions discussed above withrespect to block 1360. In an embodiment, the chrominance extractioncircuit 1515 may include an electronic hardware processor, such asprocessor 1404 of FIG. 14A or 14B. In some aspects, the chrominanceextraction circuit 1515 may comprise one or more of a processor, signalgenerator, transceiver, decoder, or a combination of hardware and/orsoftware component(s), circuits, and/or module(s).

The device 1500 further includes a demodulation circuit 1520. In anembodiment, the demodulation circuit 1520 may be configured to performone or more of the functions discussed above with respect to block 1365.In an embodiment, the demodulation circuit 1520 may include anelectronic hardware processor, such as processor 1404 of FIG. 14A or14B. In some aspects, the demodulation circuit 1520 may comprise one ormore of a processor, signal generator, transceiver, decoder, or acombination of hardware and/or software component(s), circuits, and/ormodule(s).

The device 1500 further includes a modulation circuit 1525. In anembodiment, the modulation circuit 1525 may be configured to perform oneor more of the functions discussed above with respect to block 1370. Inan embodiment, the modulation circuit 1525 may include an electronichardware processor, such as processor 1404 of FIG. 14A or 14B. In someaspects, the modulation circuit 1525 may comprise one or more of aprocessor, signal generator, transceiver, decoder, or a combination ofhardware and/or software component(s), circuits, and/or module(s).

The device 1500 further includes a luminance extraction circuit 1530. Inan embodiment, the luminance extraction circuit 1530 may be configuredto perform one or more of the functions discussed above with respect toblock 1375. In an embodiment, the luminance extraction circuit 1530 mayinclude an electronic hardware processor, such as processor 1404 of FIG.14A or 14B. In some aspects, the luminance extraction circuit 1530 maycomprise one or more of a processor, signal generator, transceiver,decoder, or a combination of hardware and/or software component(s),circuits, and/or module(s).

The device 1500 further includes an image creation circuit 1540. In anembodiment, the image creation circuit 1540 may be configured to performone or more of the functions discussed above with respect to block 1320.In an embodiment, the image creation circuit 1540 may include anelectronic hardware processor, such as processor 1404 of FIG. 14A or14B. In some aspects, the image creation circuit 1540 may comprise oneor more of a processor, signal generator, transceiver, decoder, or acombination of hardware and/or software component(s), circuits, and/ormodule(s). For example, in some aspects, the image creation circuit 1540may include the universal demosaic 1432 and the processor 1404. Forexample, the universal demosaic 1432 may generate data for the tripleplane image and send the data to the processor 1404. The processor maythen generate the image.

The technology described herein is operational with numerous othergeneral purpose or special purpose computing system environments orconfigurations. Examples of well-known computing systems, environments,and/or configurations that may be suitable for use with the inventioninclude, but are not limited to, personal computers, server computers,hand-held or laptop devices, multiprocessor systems, processor-basedsystems, programmable consumer electronics, network PCs, minicomputers,mainframe computers, distributed computing environments that include anyof the above systems or devices, and the like.

As used herein, instructions refer to computer-implemented steps forprocessing information in the system. Instructions can be implemented insoftware, firmware or hardware and include any type of programmed stepundertaken by components of the system.

A processor may be any conventional general purpose single- ormulti-chip processor such as a Pentium® processor, a Pentium® Proprocessor, a 8051 processor, a MIPS® processor, a Power PC® processor,or an Alpha® processor. In addition, the processor may be anyconventional special purpose processor such as a digital signalprocessor or a graphics processor. The processor typically hasconventional address lines, conventional data lines, and one or moreconventional control lines.

The system is comprised of various modules as discussed in detail. Ascan be appreciated by one of ordinary skill in the art, each of themodules comprises various sub-routines, procedures, definitionalstatements and macros. Each of the modules are typically separatelycompiled and linked into a single executable program. Therefore, thedescription of each of the modules is used for convenience to describethe functionality of the preferred system. Thus, the processes that areundergone by each of the modules may be arbitrarily redistributed to oneof the other modules, combined together in a single module, or madeavailable in, for example, a shareable dynamic link library.

The system may be used in connection with various operating systems suchas Linux®, UNIX® or Microsoft Windows®.

The system may be written in any conventional programming language suchas C, C++, BASIC, Pascal, or Java, and ran under a conventionaloperating system. C, C++, BASIC, Pascal, Java, and FORTRAN are industrystandard programming languages for which many commercial compilers canbe used to create executable code. The system may also be written usinginterpreted languages such as Perl, Python or Ruby.

Those of skill will further appreciate that the various illustrativelogical blocks, modules, circuits, and algorithm steps described inconnection with the embodiments disclosed herein may be implemented aselectronic hardware, computer software, or combinations of both. Toclearly illustrate this interchangeability of hardware and software,various illustrative components, blocks, modules, circuits, and stepshave been described above generally in terms of their functionality.Whether such functionality is implemented as hardware or softwaredepends upon the particular application and design constraints imposedon the overall system. Skilled artisans may implement the describedfunctionality in varying ways for each particular application, but suchimplementation decisions should not be interpreted as causing adeparture from the scope of the present disclosure.

The various illustrative logical blocks, modules, and circuits describedin connection with the embodiments disclosed herein may be implementedor performed with a general purpose processor, a digital signalprocessor (DSP), an application specific integrated circuit (ASIC), afield programmable gate array (FPGA) or other programmable logic device,discrete gate or transistor logic, discrete hardware components, or anycombination thereof designed to perform the functions described herein.A general purpose processor may be a microprocessor, but in thealternative, the processor may be any conventional processor,controller, microcontroller, or state machine. A processor may also beimplemented as a combination of computing devices, e.g., a combinationof a DSP and a microprocessor, a plurality of microprocessors, one ormore microprocessors in conjunction with a DSP core, or any other suchconfiguration.

In one or more example embodiments, the functions and methods describedmay be implemented in hardware, software, or firmware executed on aprocessor, or any combination thereof. If implemented in software, thefunctions may be stored on or transmitted over as one or moreinstructions or code on a computer-readable medium. Computer-readablemedia include both computer storage media and communication mediaincluding any medium that facilitates transfer of a computer programfrom one place to another. A storage medium may be any available mediathat can be accessed by a computer. By way of example, and notlimitation, such computer-readable media can comprise RAM, ROM, EEPROM,CD-ROM or other optical disk storage, magnetic disk storage or othermagnetic storage devices, or any other medium that can be used to carryor store desired program code in the form of instructions or datastructures and that can be accessed by a computer. Also, any connectionis properly termed a computer-readable medium. Disk and disc, as usedherein, includes compact disc (CD), laser disc, optical disc, digitalversatile disc (DVD), floppy disk and Blu-ray disc where disks usuallyreproduce data magnetically, while discs reproduce data optically withlasers. Combinations of the above should also be included within thescope of computer-readable media.

The foregoing description details certain embodiments of the systems,devices, and methods disclosed herein. It will be appreciated, however,that no matter how detailed the foregoing appears in text, the systems,devices, and methods can be practiced in many ways. As is also statedabove, it should be noted that the use of particular terminology whendescribing certain features or aspects of the invention should not betaken to imply that the terminology is being re-defined herein to berestricted to including any specific characteristics of the features oraspects of the technology with which that terminology is associated.

It will be appreciated by those skilled in the art that variousmodifications and changes may be made without departing from the scopeof the described technology. Such modifications and changes are intendedto fall within the scope of the embodiments. It will also be appreciatedby those of skill in the art that parts included in one embodiment areinterchangeable with other embodiments; one or more parts from adepicted embodiment can be included with other depicted embodiments inany combination. For example, any of the various components describedherein and/or depicted in the Figures may be combined, interchanged orexcluded from other embodiments.

With respect to the use of substantially any plural and/or singularterms herein, those having skill in the art can translate from theplural to the singular and/or from the singular to the plural as isappropriate to the context and/or application. The varioussingular/plural permutations may be expressly set forth herein for sakeof clarity.

It will be understood by those within the art that, in general, termsused herein are generally intended as “open” terms (e.g., the term“including” should be interpreted as “including but not limited to,” theterm “having” should be interpreted as “having at least,” the term“includes” should be interpreted as “includes but is not limited to,”etc.). It will be further understood by those within the art that if aspecific number of an introduced claim recitation is intended, such anintent will be explicitly recited in the claim, and in the absence ofsuch recitation no such intent is present. For example, as an aid tounderstanding, the following appended claims may contain usage of theintroductory phrases “at least one” and “one or more” to introduce claimrecitations. However, the use of such phrases should not be construed toimply that the introduction of a claim recitation by the indefinitearticles “a” or “an” limits any particular claim containing suchintroduced claim recitation to embodiments containing only one suchrecitation, even when the same claim includes the introductory phrases“one or more” or “at least one” and indefinite articles such as “a” or“an” (e.g., “a” and/or “an” should typically be interpreted to mean “atleast one” or “one or more”); the same holds true for the use ofdefinite articles used to introduce claim recitations. In addition, evenif a specific number of an introduced claim recitation is explicitlyrecited, those skilled in the art will recognize that such recitationshould typically be interpreted to mean at least the recited number(e.g., the bare recitation of “two recitations,” without othermodifiers, typically means at least two recitations, or two or morerecitations). Furthermore, in those instances where a conventionanalogous to “at least one of A, B, and C, etc.” is used, in generalsuch a construction is intended in the sense one having skill in the artwould understand the convention (e.g., “a system having at least one ofA, B, and C” would include but not be limited to systems that have Aalone, B alone, C alone, A and B together, A and C together, B and Ctogether, and/or A, B, and C together, etc.). In those instances where aconvention analogous to “at least one of A, B, or C, etc.” is used, ingeneral such a construction is intended in the sense one having skill inthe art would understand the convention (e.g., “a system having at leastone of A, B, or C” would include but not be limited to systems that haveA alone, B alone, C alone, A and B together, A and C together, B and Ctogether, and/or A, B, and C together, etc.). It will be furtherunderstood by those within the art that virtually any disjunctive wordand/or phrase presenting two or more alternative terms, whether in thedescription, claims, or drawings, should be understood to contemplatethe possibilities of including one of the terms, either of the terms, orboth terms. For example, the phrase “A or B” will be understood toinclude the possibilities of “A” or “B” or “A and B.”

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting.

What is claimed is:
 1. A demosaicing system for converting image datagenerated by an image sensor into an image, the demosaicing systemcomprising: an electronic hardware processor, configured to: determine aconfiguration of an image sensor that generated the image; determinechrominance components and luminance components of the image data basedon the configuration of the electronic image sensor; and generate theimage based on the chrominance and luminance components.
 2. Thedemosaicing system of claim 1, wherein the image sensor comprises: anarray of sensor elements; and an array of filter elements disposedadjacent to and overlaying the array of sensor elements.
 3. Thedemosaicing system of claim 2, wherein the configuration of the imagesensor comprises a length dimension and a width dimension of the filterelements in the array of filter elements.
 4. The demosaicing system ofclaim 1, wherein the electronic hardware processor is further configuredto generate a modulation function based on the configuration of theimage sensor, and demodulate the image data using the modulationfunction to determine the chrominance and luminance components.
 5. Thedemosaicing system of claim 2, wherein the configuration of the imagesensor further comprises: a period of filter elements comprising atleast one filter element, each filter element having a spectral range,the array of filter elements comprising a repeating pattern of theperiod of filter elements; and a size of each filter element having alength dimension and a width dimension that is different than arespective length dimension and a respective width dimension of acorresponding sensor element of the image sensor.
 6. The demosaicingsystem of claim 5, wherein the electronic hardware processor determinesa set of sub-lattice parameters based on the configuration of the imagesensor, the set of sub-lattice parameters comprising: a lattice matrix Mbased on a number of filter elements in the period of filter elements; aset of values comprising a spectral range of each filter element in oneperiod of the image sensor Ψ; and a set of offset vectors {B_(S)}_(SεΨ),each vector comprising a coordinate on an x y plane for a correspondingfilter element in the period of filter elements, wherein determinationof the modulation function is based on the set of sub-latticeparameters.
 7. The system of claim 4, wherein the electronic hardwareprocessor is further configured to: generate a set of configurationparameters based on the modulation function; extract a set ofchrominance components from the image data using the set ofconfiguration parameters; demodulate the chrominance components into aset of baseband chrominance components using the set of configurationparameters; modulate the set of baseband chrominance components todetermine a set of carrier frequencies; and extract a luminancecomponent from the image data using the set of carrier frequencies,wherein the image is generated based on the extracted luminancecomponent and the determined set of baseband chrominance components. 8.The demosaicing system of claim 7, wherein the set of configurationparameters comprise: a set of high pass frequency filters configured toextract the set of chrominance components from the image data based onthe configuration of the image sensor; and a set of edge detectingfilters configured to determine an energy level of the image data in atleast a horizontal direction, a vertical direction, and a diagonaldirection, the energy level indicative of an intensity difference of theradiation that is incident on neighboring sensor elements, wherein theconfiguration parameters are determined based on the modulationfunction.
 9. The demosaicing system of claim 1, wherein theconfiguration of the image sensor comprises a periodicity of a filterelement in a repeated pattern of the array of filter elements.
 10. Thedemosaicing system of claim 9, wherein the electronic hardware processoris configured to receive information indicative of the image sensorconfiguration and use the received information to determine theconfiguration of the image sensor.
 11. The demosaicing system of claim1, further comprising an electronic mobile device comprising theelectronic hardware processor.
 12. The demosaicing system of claim 1,further comprising the image sensor and an external interface, whereinthe image sensor is coupled to the electronic hardware processor andcomprises an array of sensor elements and an array of filter elements,and wherein the external interface is configured to electronicallyreceive the image sensor configuration information from another device.13. The demosaicing system of claim 1, wherein the image data comprisesa single plane spectral image, and the generated image comprises atleast one spectral plane.
 14. A method for converting image datagenerated by an image sensor into an image, the method comprising:determining a configuration of the image sensor that generated the imagedata; determining chrominance components and luminance components of theimage data based on the configuration of the electronic image sensor;and generating the image based on the chrominance and luminancecomponents.
 15. The method of claim 14, further comprising generating amodulation function based on the configuration of the image sensor, anddemodulating the image data using the modulation function to determinethe chrominance and luminance components.
 16. The method of claim 15,wherein the configuration of the image sensor is defined by: a periodicarrangement of filter elements, each filter element comprising aspectral range; and a size of each filter element having a lengthdimension and a width dimension that is different than a respectivelength dimension and a respective width dimension of a correspondingsensor element of the image sensor.
 17. The method of claim 15, furthercomprising: generating a set of configuration parameters based on thedetermined modulation function; extracting a set of chrominancecomponents from the image data using the set of configurationparameters; demodulating the set of chrominance components into a set ofbaseband chrominance components using the set of configurationparameters; modulating the set of baseband chrominance components todetermine a set of carrier frequencies; and extracting a luminancecomponent from the image data using the set of carrier frequencies,wherein the generation of the image is based on the extracted luminancecomponent and the set of baseband chrominance components.
 18. The methodof claim 17, wherein the set of configuration parameters comprises: aset of high pass frequency filters configured to extract the set ofchrominance components from the image data based on the configuration ofthe image sensor; and a set of edge detecting filters configured todetermine an energy level of the image data in at least a horizontaldirection, a vertical direction, and a diagonal direction, the energylevel indicative of an intensity difference of the radiation that isincident on neighboring sensor elements, wherein the configurationparameters are determined based on the modulation function.
 19. Themethod of claim 15, further comprising determining a set of sub-latticeparameters based on the configuration of the image sensor, the set ofsub-lattice parameters comprising: a lattice matrix M based on a numberof filter elements in the period of filter elements; a set of valuescomprising a spectral range of each filter element in one period of theimage sensor Ψ; and a set of offset vectors {B_(S)}_(SεΨ), each vectorcomprising a coordinate on an x y plane for a corresponding filterelement in the period of filter elements, wherein determination of themodulation function is based on the set of sub-lattice parameters.
 20. Anon-transitory computer-readable medium comprising code that, whenexecuted, causes an electronic hardware processor to perform a method ofconverting image data generated by an image sensor into a second image,the method comprising: determining a configuration of the image sensorthat generated the image data; determining chrominance components andluminance components of the image data based on the configuration of theelectronic image sensor; and generating the image based on thechrominance and luminance components.